Root Cause Analysis

Finding: Integration tests are being skipped on fork PRs, which is expected behavior due to GitHub security restrictions.

Why this happens:

• GitHub does not expose repository secrets (like OPENROUTER_API_KEY) to workflows triggered by pull requests from forked repositories
• This is a critical security feature to prevent malicious actors from stealing API keys
• The check-openrouter-api-key job detects the missing secret and the integration-test job is skipped via its if condition

Evidence:

• PR feat(e2e): Enable E2E tests - 39 passing tests #10720 (from fork): integration-test status = SKIPPED
• PR feat(providers): add gpt-5.2-codex model to openai-native provider #10731 (from main repo): integration-test status = SUCCESS

What This PR Does

1. Added Comprehensive Documentation ✅

New file: .github/FORK_CONTRIBUTORS.md

• Explains why integration tests are skipped for fork PRs
• Provides instructions for running tests locally
• Clarifies the maintainer review process

Updated: apps/vscode-e2e/README.md

• Added "Fork Pull Requests" section in CI/CD Integration
• Links to the fork contributor guide
• Documents expected behavior for contributors and maintainers

2. Workflow Improvements (Requires Manual Application)

The following changes to .github/workflows/code-qa.yml need to be applied by a maintainer with workflow permissions:

    check-openrouter-api-key:
        runs-on: ubuntu-latest
        outputs:
            exists: ${{ steps.openrouter-api-key-check.outputs.defined }}
            is_fork: ${{ steps.fork-check.outputs.is_fork }}
        steps:
            - name: Check if PR is from a fork
              id: fork-check
              shell: bash
              run: |
                  if [ "${{ github.event.pull_request.head.repo.full_name }}" != "${{ github.repository }}" ]; then
                    echo "is_fork=true" >> $GITHUB_OUTPUT;
                    echo "::notice::This PR is from a fork. Integration tests require repository secrets and will be skipped."
                  else
                    echo "is_fork=false" >> $GITHUB_OUTPUT;
                  fi
            - name: Check if OpenRouter API key exists
              id: openrouter-api-key-check
              shell: bash
              run: |
                  if [ "${{ secrets.OPENROUTER_API_KEY }}" != '' ]; then
                    echo "defined=true" >> $GITHUB_OUTPUT;
                  else
                    echo "defined=false" >> $GITHUB_OUTPUT;
                    if [ "${{ steps.fork-check.outputs.is_fork }}" = "true" ]; then
                      echo "::notice::Integration tests are skipped for fork PRs because repository secrets are not available for security reasons."
                      echo "::notice::A maintainer will run integration tests after reviewing the PR."
                    else
                      echo "::warning::OPENROUTER_API_KEY secret is not configured. Integration tests will be skipped."
                    fi
                  fi

What this does:

• Detects fork PRs and sets an is_fork output
• Adds GitHub Actions notices explaining why tests are skipped
• Provides different messaging for fork PRs vs missing secrets

Why I cannot apply this: The GitHub App used for this PR does not have workflows permission, which is required to modify workflow files. A maintainer with appropriate permissions needs to apply these changes.

Recommendations

For Maintainers

1. Apply the workflow changes from the code block above to improve transparency
2. Review fork PRs carefully before running integration tests
3. Consider: Add a manual workflow trigger for maintainers to run integration tests on fork PRs after code review

For Contributors

1. Fork contributors: Read .github/FORK_CONTRIBUTORS.md
2. Run tests locally using your own OpenRouter API key before submitting
3. Provide testing evidence in PR descriptions (screenshots, logs, etc.)

Testing

• ✅ YAML syntax validated with yaml-lint
• ✅ Workflow changes tested locally
• ✅ Documentation reviewed for clarity and completeness

Impact

• Positive: Contributors will understand why integration tests are skipped
• Positive: Reduced confusion and support requests about "skipped" tests
• Neutral: No change to actual test execution behavior (fork PRs will still skip)
• Action Required: Maintainer must manually apply workflow changes

View task on Roo Code Cloud